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The  Honorable  Donald  J.  Devine 
Director,  Office  of  Personnel 
Management 

Dear  Dr.  Devine: 
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Sub jectfvybb3tacles  Hamper  the  Office  of  Personnel 

Management's  Evaluation  of  the  Implementa¬ 
tion  of  the  1978  Civil  Service  Reform  Act» 
(PPCD-81-69) 

We  have  recently  completed  a  limited  study  of  the  Office 
of  Personnel  Management's  (0PM* s)  5~year  strategy  for  evalu¬ 
ating  the  implementation  and  impact  cf  the  Civil  Service  Re¬ 
form  Act  of  1978  (CSRA) .  In  that  study,  we  identified  prob¬ 
lems  which,  we  believe,  could  hamper  the  collection  of  data 
needed  to  evaluate  CSRA.  We  are  therefore  advising  you  of 
our  findings  to  assist  you  in  your  consideration  of  the 
evaluation  strategy. 

We  identified  the  following  management  and  technical 
weaknesses  which,  we  believe,  warrant  your  careful  consider¬ 
ation. 

— The  basic  objectives  of  the  evaluation  are  not  clearly 
defined. 

— The  effective  central  management  of  the  evaluation's 
activities  and  resources  is  hampered. 

— The  design  and  implementation  of  the  evaluation  have 
technical  limitations. 

In  view  of  these  weaknesses,  we  are  concerned  whether  the 
ongoing  and  planned  series  of  studies  will  provide  the  infor¬ 
mation  anticipated  by  the  evaluation  strategy. 
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OBJECTIVES,  SCOPE,  AND  METHODS  GY 

In  conducting  our  limited  review,  we  studied  the 
evaluation  strategy  package,  interviewed  officials  in  the 
Office  of  Planning  and  Evaluation/CSRA  Evaluation  Manage¬ 
ment  Division  (EMD)  and  the  Office  of  Agency  Compliance 
and  Evaluation,  and  reviewed  documentation  on  the  perform¬ 
ance  appraisal  and  merit  pay  case  and  special  studies. 

We  also  reviewed  the  proposals  and  the  first-year  draft 
reports  of  the  university  researchers  performing  the  organ¬ 
izational  assessments  and  discussed  the  work  with  one  of 
the  principal  researchers. 

OBJECTIVES  SHOULD  BE  DEFINED  AND  EMPHASIZED 

The  evaluation's  overall  objectives  are  not  precisely 
defined  in  writing  by  OPM.  The  Acting  Chief,  EMD,  defined 
the  objectives  generally  as  a  series  of  long-term  studies 
to  determine  if  CSRA's  goals  are  being  met.  Without  a  more 
precise  agency  statement  of  the  objectives  and  a  plan  for 
synthesizing  and  reporting  data  from  the  various  sources, 
we  could  not  determine  how  the  accumulated  information  will 
be  used  or  how  valuable  it  may  be  to  the  Congress,  OPM,  or 
other  agencies. 

Although  an  earlier  draft  of  the  strategy  package 
contained  an  objective  statement,  the  present  strategy  pack¬ 
age  does  not  contain  a  statement  of  the  overall  objectives 
of  the  evaluation.  From  the  strategy  package  and  from  dis¬ 
cussions  with  OPM  officials,  we  inferred  that  the  objectives 
were  to  examine 

—  the  effects  of  Government-wide  implementation  of 
specific  civil  service  reform  initiatives, 

—  the  effects  of  implementing  single  reform  initi¬ 
atives  in  individual  agencies,  and 

— the  collective  effects  of  CSRA. 

The  need  for  a  clear  statement  of  OPM's  objectives 
becomes  more  apparent  when  considering  the  complexity  of 
the  evaluation  and  the  involvement  of  various  OPM  groups 
in  the  series  of  studies.  The  OPM  strategy  for  evaluating 
civil  service  reform  consists  of  (1)  program  plans  for 
evaluating  11  individual  reform  initiatives,  such  as  the 
Senior  Executive  Service,  the  Merit  Pay  Program,  and 
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the  new  performance  appraisal  system,  (2)  an  integrated 
examination  of  selected  reforms  by  independent  contractors, 
and  (3)  a  survey  of  Federal  employees'  attitudes  about  the 
reforms  and  their  effects. 

The  program  plans  for  evaluating  the  11  individual 
initiatives  call  for  the  use  of  a  combination  of  case  studies, 
special  studies,  and  management  information  generated  by  OPM 
program  offices  responsible  for  monitoring  implementation  of 
the  reforms  by  Federal  agencies.  In  addition,  the  evaluation 
of  individual  reforms  will  draw  on  survey  information  and  the 
contractors'  work.  Practically  every  major  operating  group  in 
OPM  is  involved  in  the  5-year  evaluation  as  either  participants 
in  studies  or  sources  of  data.  Some  groups  are  also  making 
their  own  short-term  assessments  of  the  CSRA  implementation. 

According  to  the  Acting  Chief,  EMD,  separate  reports  for 
each  of  the  major  studies  will  be  prepared  jointly  by  EMD  and 
the  program  offices  responsible  for  carrying  out  the  studies. 
EMD  plans  to  then  periodically  issue  reports  consolidating 
these  individual  study  efforts.  The  first  summary  report  is 
scheduled  for  the  end  of  calendar  year  1981.  However,  it  is 
not  clear  to  us  how  OPM  plans  to  tie  the  accumulated  data  to 
the  questions  contained  in  the  strategy  package. 

OBSTACLES  TO  EFFECTIVE  CENTRAL  MANAGEMENT 


Effective  central  management  of  OPM' a  evaluation  effort 
is  hampered  by 

— overlapping  authorities  in  the  design,  execution, 
and  reporting  for  the  specific  projects  and 

— competing  resource  demands  of  the  program  offices. 

The  OPM  offices  participating  in  the  evaluation  should  have  a 
clear  understanding  of  their  responsibilities  and  the  1  mits  of 
their  authorities.  In  our  opinion,  the  resource  demands  of 
short-term  assessments  may  adversely  affect  the  resources 
available  for  the  long-term  studies  especially  during  staff 
shortage  periods. 

Overlapping  authorities 

The  authorities  and  roles  of  OPM  offices  contributing 
to  the  5-year  evaluation  are  not  clearly  defined.  Division 
of  responsibilities  for  project  work  segments  is  reached 
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by  group  consensus  among  EMD  and  the  participating  program 
groups.  Shared  authority  for  project  design,  execution,  and 
reporting  has  resulted  in  delays. 

To  meet  deadlines  imposed  on  the  various  studies  in  the 
5-year  evaluation,  EMD  must  rely  on  appeals  to  the  program 
offices  to  honor  their  commitments.  The  Acting  Chief  said 
that  EMD's  close  involvement  in  the  day-to-day  progress 
of  ongoing  evaluation  projects  is  limited  by  the  small  staff 
assigned  to  EMD.  EMD  is  now  implementing  a  computerized 
tracking  system  which  will  generate  quarterly  status  reports 
on  the  internal  CSRA  evaluation  activities.  The  data  summary 
sheets,  used  for  coding  information  into  the  system,  will 
serve  as  commitments  from  the  participating  groups  to  meet 
milestones  and  to  deliver  end  products. 

Competing  resource  demands 

Program  offices  have  a  major  role  in  collecting  data  for 
segments  of  the  long-term  evaluation.  However,  their  primary 
responsibilities  involve  providing  technical  assistance  and 
guidance  to  the  agencies  implementing  the  CSRA  initiatives. 

In  carrying  out  those  responsibilities,  the  offices  are  con¬ 
cerned  with  problem  identification  and  resolution  to  promote 
program  implementation.  Because  the  5-year  evaluation  is 
not  designed  to  provide  short-term  assessment  information  on 
implementation  problems,  program  offices  must  therefore  ini¬ 
tiate  their  own  studies  to  meet  these  needs. 

Since  resources  needed  for  short-  and  long-term 
evaluation  are  taken  from  the  same  pool,  long-term  evaluation 
projects  may  be  delayed  by  the  competition  for  limited  re¬ 
sources  during  staff  shortage  periods.  For  example,  analysis 
of  the  data  collected  fox.  the  performance  appraisal  special 
study  was  delayed  because  of  staff  shortages  and  the  priority 
of  other  work.  Similar  situations  could  be  avoided  if  the 
program  offices'  Bhort-term  needs  were  recognized  and  steps 
were  taken  to  avoid  the  competition  for  resources. 

TECHNICAL  LIMITATIONS  IN 
DESIGN  AND  IMPLEMENTATION 


The  following  technical  problems  in  the  evaluation 
studies  may  affect  the  data's  usefulness  and  validity. 

— Information  developed  by  most  of  the  evaluation  methods 
cannot  be  generalized  beyond  the  specific  observation 
sites-  The  major  exceptions  are  the  Federal  employees' 
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attitude  survey  and  some  program  information  systems  which 
will  generate  Government-wide  data. 

— The  evaluation  strategy  appears  to  focus  on  the  collection 
of  data  which  may  provide  inadequate  support  for  conclu¬ 
sions  about  CSRA's  implementation. 

Three  major  methodologies — organizational  assessments, 
case  studies,  1/  and  special  studies — are  not  designed  to 
provide  data  that  can  be  projected  beyond  the  specific  work 
sites.  These  studies  are  designed  as  onsite  evaluations  of 
varying  depths  to  assess  the  implementation  of  different  CSRA 
provisions.  In  an  organizational  assessment  draft  report,  a 
contractor  cautioned  against  the  comparative  use  of  reports 
for  the  five  sites  studied  and  stressed  the  individuality  of 
the  data  collection  at  each  site.  Office  of  Agency  Compliance 
and  Evaluation  officials  also  said  that,  in  most  instances, 
data  from  the  special  or  case  studies  cannot  be  generalized. 
The  General  Schedule  Classification,  Grade  and  Pay  Retention 
Special  Study  is  the  only  special  or  case  study  designed  to 
provide  Government-wide  information. 

The  major  emphasis  of  the  studies  seems  to  be  placed  on 
the  frequency  of  events  at  specific  sites  and  on  attitudinal 
change.  It  is  not  clear  how  these  studies  will  collectively 
generate  data  needed  to  (1)  support  conclusions  about  the 
Government-wide  implementation  of  civil  service  reform  initi¬ 
atives  or  (2)  serve  as  a  basis  for  recommending  legislative 
or  policy  changes. 

Now  that  some  of  the  first- round  studies  have  been 
completed  and  are  in  the  analysis  and  report-writing  phases, 
the  developed  data  should  be  scrutinized  to  determine  if  it 
is  the  type  and  quality  required  to  answer  the  questions 
posed  in  the  strategy  package  about  civil  service  reform. 


The  issues  discussed  in  this  letter  may  not  be  a  complete 
list  of  the  obstacles  that  could  be  encounted  in  the  CSRA  eval¬ 
uation.  However,  we  are  advising  you  of  our  findings  to  assist 
you  in  assessing  this  project. 


l_/According  to  the  Acting  Chief,  EMD,  case  studies  conducted 
by  the  Office  of  Agency  Compliance  and  Evaluation  were  to 
be  discontinued  as  of  August  18,  1981. 
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We  would  appreciate  being  advised  of  any  action  you  plan 
to  take  as  a  result  of  the  issues  we  have  raised  in  this 
letter  and  would  be  happy  to  meet  with  you  and  your  staff 
to  discuss  these  matters  further  if  you  wish. 

Because  of  the  interest  various  congressional  committees 
have  in  civil  service  reform,  wo  are  sending  copies  of  this 
letter  to  the  Chairmen,  House  Committee  on  Post  Office  and 
Civil  Service  and  House  Subcommittee  on  Civil  Service,  and 
to  the  Chairmen,  Senate  Committee  on  Governmental  Affairs 
and  Senate  Subcommittee  on  Civil  Service  and  General  Studies. 

Sincerely  yours, 

Chiajrbrd  I.  Gould 

Director 
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